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Abstract  of 

OPERATIONAL  NET  ASSESSMENT: 

A  Framework  for  Social  Network  Analysis  and  Requirements  for  Critical  Debate 


Operational  Net  Assessment  (ONA)  is  critical  to  the  new  Standing  Joint  Force 
Headquarters  (SJFHQ)  concept.  SJFHQs  were  designed  by  the  US  Joint  Forces  Command 
(JFCOM)  to  provide  Regional  Combatant  Commanders  in-depth  analysis  for  a  specific 
adversary  or  situation  within  their  area  of  responsibility.  This  paper  does  not  seek  to  debunk 
the  ONA  process,  nor  antagonize  the  SJFHQ  concept.  The  author  desires  to  provide  insight 
upon  perceived  weaknesses  in  existing  doctrine  and  promote  issues  for  further  discussion 
within  the  Joint  Intelligence  Community.  The  collaborative  ONA  process  as  designed  by 
JFCOM  is  subjective.  The  lack  of  a  formal  vetting  process  to  review  information  applied  for 
analysis  is  dangerous.  Current  network  modeling  tools,  the  cornerstone  of  ONA’s  analytical 
construct,  have  limitations  that  must  be  recognized  as  tenuous  assumptions.  Product  output 
must  be  sufficiently  analyzed  in  concert  with  operational  planners  and  scaled  to  mission 
requirements  to  support  the  Commander’s  Intent.  The  author  has  drawn  from  current 
literature  on  the  ONA  template  and  reviewed  the  construct  in  order  to  create  a  “truth  in 
lending”  approach.  The  goal  is  not  simply  to  identify  the  present  limitations  of  ONA,  but 
provide  recommendations  and  areas  for  improvement.  For  ONA  to  be  relevant,  its  level  of 
confidence  must  be  clearly  understood  by  the  warfighter. 
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Introduction 


The  difficulty  of  accurate  recognition  constitutes  one  of  the  most  serious  sources  of 

friction  in  war. 1 

—  Carl  von  Clausewitz 

While  the  doctrinal  explanations  of  Operational  Net  Assessment  (ONA)  appear  rather 
straightforward,  there  is  a  definite  focus  on  results  (ends)  to  the  exclusion  of  process  (ways  or 
means).  Discussion  of  potential  bias  within  information  or  analyst  perceptions  is  missing  and 
must  be  accountable  in  the  process.  Human  nature  prevents  total  objectivity:  “'The  process 
of  intelligence  analysis  and  assessment  is  a  very  personal  one.  There  is  no  agreed-upon 
analytical  schema,  and  the  analyst  must  use  his  belief  system  to  make  assumptions  and 
interpret  information. As  Robert  Deutsch  has  noted  about  American  culture:  “Attempts  at 
image  creation  are  now  an  invasive  part  of  our  environment:  some  pollute  and  some  enhance 
human  experience.”  This  applies  as  well  to  the  analytical  construct  of  ONA.  Whether  the 
image  created  is  driven  by  the  ONA  element  itself,  information  provided  by  outside 
Agencies,  or  the  way  we  apply  modem  technology;  all  of  our  limitations  must  be  observed 
and  understood.  Uncertainty  must  be  factored  in  a  realistic  process  for  ONA.  The  level  of 
confidence  in  the  analysis  must  be  a  core  component  of  the  end  product  for  the  warfighter. 

We  must  be  always  wary  of  the  “hard  facts  of  capability  and  the  soft  assumptions  of 
intention.”4 

ONA  and  the  SJFHQ:  Background  and  Definitions 
The  secret  of  a  sound,  satisfactory  decision... has  always  been  that  the  responsible 
official  has  been  living  with  the  problem  before  it  becomes  acute.5 


—  President  Dwight  D.  Eisenhower 


Joint  Forces  Command  (JFCOM)’s  doctrine  declares  ONA  as  “the  integration  of 
people,  processes,  and  tools  that  use  multiple  information  sources  and  collaborative  analysis 
to  build  shared  knowledge  of  the  adversary,  the  environment,  and  ourselves.”6  ONA’s  ability 
to  predict  adversary  actions  as  resultant  effects  from  our  own  efforts  is  the  ultimate  goal  in 
supporting  Effects-Based  Operations  (EBO).  Doctrine  explains  this  as  a  long-term  analytical 
process  where  the  Standing  Joint  Force  Headquarters  (SJFHQ)  and  its  ONA  element  delve 
into  a  Commander’s  prioritized  regional  concerns  long  before  a  crisis  brews. 

Current  literature  frames  ONA  as  interpreting  significance  from  an  adversary  through 
the  lens  of  systems.7  A  critical  portion  of  ONA  is  System-of-Systems  Analysis  (SoSA), 
which  seeks  to  “identify,  analyze,  and  relate  the  goals  and  objectives,  organization, 
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dependencies...  inter-dependencies  [and]  influences”  of  an  adversary  under  investigation. 

The  linkage  between  SoSA  and  EBO  is  through  determining  vulnerabilities  of  the  adversary. 
This,  in  turn,  creates  exploitation  possibilities  for  desired  effects,  determined  by  the 
Commander.9  Per  JFCOM  doctrine,  the  SoSA  process  is  heavily  reliant  upon  information 
provided  to  the  SJFHQ ’s  ONA  team  by  groups  both  within  the  US  Government  and  on  the 
outside.  Non-go vemmental  organizations  (NGOs)  are  listed  as  core  elements  of  ONA  input, 
along  with  Centers  of  Excellence  (COE):  Academic  institutions,  laboratories,  and  think 
tanks.10  The  SoSA  approach  dissects  the  adversary  into  smaller  functional  components  for 
individual  study.11  SoSA  measures  causal  relationships  between  various  Blue  actions  and 
subsequent  Red  effects  within  the  elements  of  national  power:  Diplomatic,  Infonnation, 
Military,  and  Economic  (DIME).  "  The  analysis  is  conducted  against  six  areas  of  an 
adversary’s  capabilities  or  organization  (political,  military,  economic,  social,  information, 
and  infrastructure  -  PMESII).13 
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Data  Requirements  for  ON  A 


We  could  have  talked  about  the  science  of  Intelligence,  but... the  science  of 
Intelligence  is  yet  to  be  invented. 14 

—  Charles  Allen 

The  crux  of  SoSA  is  inherent  in  its  name:  the  study  of  systems.  “A  mature  ONA  for  a 
single  focus  area  will  likely  entail  thousands  of  nodes  and  associated  relationships,  tasks,  and 
potential  effects.”15  The  voluminous  data  that  can  be  compiled  for  analysis  in  a  network 
construct  requires  sophisticated  technical  assistance  through  computer  simulation  modeling. 
The  two  most  basic  models  for  such  analysis  are  conceptual  and  computational.  Conceptual 
models  review  qualitative  social  and  cultural  behavior  and  explain  relationship  dynamics 
among  nodes  or  individuals.  Computational  models  provide  quantitative  input/output  results, 
normally  in  a  numeric  or  mapped  fashion.16  Both  models  allow  one  to  understand  complex 
networks  and  assist  in  developing  strategies  to  counter  adversary  actions.17  The  limitation  of 
conceptual  models  is  their  development  involves  a  long-term  research  effort.  There  are 
applications  for  such  tools  in  pre-crisis  ONA  analysis;  however,  strict  reliance  on  conceptual 
modeling  is  a  negative  for  crisis  action  planning.  The  need  to  transition  data  sets  from  a 
static  (but  robust)  conceptual  model  to  a  more  dynamic  (and  rapid)  computational  effort  has 
been  identified,  but  the  level  of  effort  weighted  in  the  computational  sphere  limits  social  and 
cultural  nuances  from  entering  the  equation. 

The  tools  available  now  cannot  handle  both  types  of  information  at  a  fidelity  required 
18 

by  ONA.  Current  systems  can  map  any  given  relationship  into  a  social  network;  provide 
the  raw  data,  and  one  has  a  wire-diagram  for  a  PowerPoint  slide  in  short  order.  However,  the 
layers  of  detail  gathered  by  the  humanist  through  conceptual  processes  are  lost.  This  skews 
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true  effects  determination,  the  reason  for  ONA  within  the  EBO  construct.  The  ability  to 
focus  on  conceptual  processes  in  aggregate  while  allowing  technology  to  assist  is  required. 
The  focus  should  be  upon  a  “’concept-driven  activity  rather  than  an  external  data-driven 
activity.’”19  The  SoSA  process  cannot  be  mutually  exclusive. 

Post-conflict  operations  in  Iraq  show  us  the  warfighter  requires  analysis  to  explain  the 
following: 

•  What  is  the  enemy  thinking  and  why? 

•  What  will  they  do  if  I  take  action  X  and  why?20 

These  are  not  pure  quantitative  expressions.  Qualitative  variables  such  as  culture  are 
significant  factors.'  To  reduce  the  time  requirements  for  this  kind  of  analysis,  software 
manufacturers  are  beginning  to  explore  this  niche. 

There  are  various  examples  of  programs  created  within  the  last  few  years  specifically 
targeting  social  network  analysis  (SNA).'  Simulation  designers  have  addressed  the  need  to 
plug  in  rule  sets  derived  from  conceptual  modeling.  This  can  be  accomplished  by  translating 
conceptual-derived  data  into  computational  algorithms  and  programmable  agents  in  a 
synthetic  environment,  so  the  conceptual  model  (and  its  social  fabric  infonnation)  is 
embedded  in  the  procedures.23  Although  this  capability  is  assumed  by  JFCOM  in  doctrinal 
ONA  publications,  the  technology  is  not  yet  there.  Owen  Cate,  the  Assistant  Director  of 
Security  Studies  Program  at  MIT,  lauds  the  continuing  research  into  SNA  advances,  but 
notes: 

I  think  it’s  one  of  these  cases  when  all  the  methodology,  all  the  fancy  software  and  all 
the  other  stuff  —  if  it’s  garbage  in,  it’s  going  to  be  garbage  out,  so  the  question  boils 
down  to  how  much  do  we  know  about  these  groups. . . [i]f  we  don’t  know  much  about 

24 

these  groups,  then  I  don’t  think  these  models  will  have  much  utility. 


4 


While  Cate’s  statement  can  be  viewed  as  outwardly  negative,  his  point  does  support  the  need 
for  integrated  conceptual,  humanistic,  and  cultural  knowledge  applied  within  any  SNA 
simulation  tool. 

The  “ONA  brochure”  glosses  over  current  limitations  and  imparts  an  almost  infallible 
capability:  .  [ONA  provides]  pertinent  expertise  and  information  for  holistic  analysis 

[emphasis  mine]  of  adversaries  and  the  potential  effects  operations  might  have  on  them.”25 
The  issue  remains  that  “current  technologies  cannot  account  for  behavior  related  to  the  social 
or  political  context.”  This  shortfall  was  noted  by  the  Information  Operations  personnel 
engaged  in  the  Millennium  Challenge  2002  (MC02)  exercise:  “Inadequate  resources  existed 
for  producing...  integration  of  cultural  intelligence,  psychological  operations,  public  affairs, 

97 

and  civil  affairs”  into  simulation  models. “ 

Future  simulation  and  modeling  systems  must  pull  in  these  disparate  variables.  Dr. 
Kathleen  Carley  of  Carnegie  Mellon  University,  a  leading  researcher  for  next-generation 
SNA  systems,  is  also  concerned: 

At  the  theoretical  level,  little  is  known  about  individual  differences  in  balancing 
social,  political,  and  group  level  concerns  and  goals.  At  the  empirical  level,  the 

validity,  collection,  and  bias  issues. .  .are  distinct  and  little  is  known  about  how  to 

28 

calibrate  data  across  levels." 

The  assumption  that  current  off-the-shelf  nodal  analysis  tools  can  provide  “complete, 
accurate  data”  is  precarious.  Missing  and  erroneous  information  must  be  scrutinized.  ONA 
doctrine  lacks  discussion  on  any  process  for  vetting  such  information,  nor  any  Quality 
Assurance  measures.  Understanding  the  limitations  of  data  input  must  be  addressed  to  shape 
the  boundaries  of  resultant  computations. 
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Understanding  Data:  Quality,  Quantity,  and  Value 
One  should  never  use  elaborate  scientific  guidelines 

30 

as  if  they  were  a  kind  of  truth  machine.  ” 

—  Carl  von  Clausewitz 

In  any  computational  model,  validity  of  information  must  be  calculated  or  weighed. 
Analysts  must  identify  the  data  as  “valid  for  whom?”  This  is  especially  true  in  calculating 
metrics  of  success.  ONA  doctrine  expresses  the  measurement  of  interagency  and  COE 
entities  “working  better  together”  as  such  a  validation  metric.  Some  may  argue  the  amount 
of  data  provided  for  SoSA  or  level  of  outside  organization  participation  is  relevant.  In 
reality,  quality  assurance  of  information  analyzed  and  prepared  for  dissemination  should  be  a 
significant  level  of  effort  within  ONA. 

This  is  a  difficult  process  for  SNA.  Traditional  analytic  tools  are  “data  greedy”:  Very 
detailed  information  is  required  to  establish  nodal  understanding  and  rudimentary 
relationships.  When  one  contemplates  shifting  analysis  from  static  to  dynamic  networks 
(such  as  terrorist  organizations  or  economic  agents),  data  requirements  become  even  more 
demanding  and  the  focus  of  the  ONA  organization  could  slide  precariously  to  quantity  of 
input  in  order  to  “keep  up”  with  the  changes.  ONA  analysts  must  resist  the  desire  to  create 
the  largest  string  of  data,  but  instead  focus  on  data  selection.  “[An]  effect  of  pushing 
intelligence  down  the  road  of  science  is  the  tendency  to  view  quantifiable  capabilities  as 
more  accurate  and  also  more  important  then  qualitative  intentions.”34 

SoSA  modeling  of  PMESII  systems  becomes  fixated  on  generating  rapid  calculations 
and  compiling  the  requisite  amount  of  data.  ~  As  SoSA  is  about  crafting  intelligence  and  not 
simply  generating  information,  the  pitfalls  of  observing  an  adversary  through  a  faulty 
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spyglass  remain/  What  you  interpret  may  be,  in  fact,  exactly  what  your  opposite  number 
wants  you  to  perceive.  “In  the  case  of  intelligence  analysis,  deception  is  the  rule. .  .the 

'XI 

validity  of  the  data  is  always  in  doubt.” 

An  example  is  provided  by  the  Joint  Forces  Intelligence  Command  (JFIC).  One  goal 

of  JFIC  is  to  reduce  the  amount  of  time  exhausted  on  collecting  and  processing  information 

in  order  to  focus  weight  of  effort  upon  analysis.  Their  metric  is  speed  of  information  input: 

“For  a  system-of-system  analysis,  Experimentation  Directorate  personnel  [at  JFCOM] 
said  it  would  take  six  man-years  to  conduct  the  nodal  analysis  for  a  specific  country. 
We  took  a  tool  and  did  the  same  search  that  identified  all  the  nodes  in  20  hours.  That 
was  just  the  beginning  of  the  research,  but  there  are  tools  out  there  to  get  through  all 
of  the  front  end  of  the  data.”38 

This  example  does  present  an  excellent  ability  to  rapidly  focus  on  the  “exploitation”  phase  of 

the  Process,  Exploitation,  and  Dissemination  (PED)  intelligence  cycle.  Analysts  are  allowed 

more  time  to  review  the  simulation  output;  however,  the  mantra  of  “trust,  but  verify”  cannot 

be  dismissed.  If  analysis  does  not  begin  until  the  initial  simulation  runs  are  complete,  how 

much  error  (or  deception)  have  you  already  absorbed? 

Some  may  argue  the  concept  of  reducing  all  adversary  mechanisms  to  a  network 

model  is  the  most  effective  procedure  to  create  rapid,  computational  products  through  SNA 

modeling.  Cognitive,  conceptual  analysis  takes  time  and  narrative  research  does  not  translate 

into  quick  action.  In  a  crisis  situation,  a  purely  qualitative  approach  would  be  detrimental. 

However,  boiling  all  of  an  adversary’s  relationships  or  organizations  down  to  a  network 

“cookie  cutter”  may  not  be  beneficial  and  can  be  a  square-peg-in-round-hole  situation. 

While  systems  of  interest  may  be  presented  as  networks, 

Many  (particularly  economic,  social  and  political  systems)  may  also  be  usefully 
represented  other  ways,  for  example  as  hierarchies/organizations,  small  group 
decision-making  bodies,  individuals  engaged  in  bargaining. .  .collective  action. . . [and 
all]  subject  to  social  and  cognitive  biases. 
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DARPA,  in  its  design  work  for  the  Integrated  Battle  Command  system,  is  similarly 
concerned  that  “conventional  models  may  not  be  relevant”  for  all  adversary  dynamics  40 
Black  markets  within  an  economy,  illegal  imports  and  exports,  social  demographics,  physical 
and  political  structural  changes  can  all  affect  our  ability  to  detennine  cause.41  This 
discussion  is  particularly  relevant  when  a  JTF  is  involved  in  Security  and  Stability 
Operations  (SASO)  or  Flexible  Deterrent  Operations.  During  these  conditions,  influence  and 
not  destruction  is  the  prime  directive.  In  these  situations,  a  network  model  would  need  to 
weigh  values  based  on  social  and  conceptual  information. 

This  is  a  challenge,  as  cultural  factor  weights  are  normally  very  difficult  to  ascertain 
and  apply  some  level  of  subjectivity.  Priorities  within  the  relationship  are  assigned  based 
on  observed  or  assessed  organizational  decision  points.43  Because  ONA  drives  a  network¬ 
mapping  focus,  some  may  argue  that  SoSA  is  to  simply  “connect  the  dots  [and]  isolate  the 
key  actors  who  are  often  defined  in  terms  of  their  ‘centrality’  to  the  network.”44  The 
weakness  is  nodal  analysis  cannot  be  taken  strictly  at  face  value.  “[Rjemoval  of  the  most 
central  node  might  leave  a  network  less  vulnerable  than  removal  of  an  emergent  leader,” 
when  applying  against  an  organization,  such  as  a  terrorist  group.45  Nodes  and  ties  resulting 
from  a  simulation  are  influenced  by  the  inherent  biases  obtained  by  a  given  sampling 
procedure  46  The  model  (or  the  analyst)  can  over-or-under-sample  certain  types  of  relations, 
which  in  output  will  “strategically  misreport’ ’  specific  ties  and  links.47 

As  ONA  and  SoSA  capabilities  mature,  they  must  be  linked  to  improved  SNA  model 
simulations  that  take  into  account  the  dynamic,  cognitive  data  faced  throughout  the  spectrum 
of  military  tasks,  not  just  higher-level  war-making.  Models  need  to  provide  a  “significant 
degree  of  irreducible  uncertainty  associated  with  the  psychological,  inter-personal,  and 
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bureaucratic  processes  within  future  US  adversaries. . . Today’s  prism  of  globalization, 
failed  states,  and  economic  change  all  point  to  increased  uncertainty.  Therefore,  not  only 
must  newer  generation  SNA  simulation  models  factor  these  scenarios,  but  ONA  analysts  and 
intelligence  professionals  must  also  operate  under  a  scalable  threshold  of  uncertainty  and 
gaps  in  order  to  continue  production  efforts  and  remain  relevant  to  the  warfighter. 

Potential  for  Bias  and  Error 

The  facts  are  mugged  long  before  they  reach  decision-makers .49 

—  Alexander  Butterfield 

One  must  be  aware  of  the  propensity  for  either  deliberate  or  unknown  subjectivity  of 
information  provided  to  SoSA  analysts.  The  focus  of  a  particular  Subject  Matter  Expert 
(SME)  involved  could  be  compartmentalized  vis-a-vis  the  broader  perspective  of  a  problem. 
Network  dynamics  under  observation  can  be  “based  on  the  estimates  of  Subject  Matter 
Experts  using  largely  the  subjective  [emphasis  original]  high-level  data  often  narrowed  to  a 
particular  area  of  expertise.”50  Subjectivity  can  increase  through  cultural  lens  discrepancies 
within  our  own  (Blue)  sphere.  That  cultural  differences  exist  to  a  certain  degree  between 
military  services  within  DoD  is  a  given;  however,  the  differences  between  governmental 
agencies  are  vast,  and  those  outside  of  government  are  even  further  removed. 

One  agency’s  view  of  mission,  legal  definitions,  and  constraints  may  all  vary  from 
that  of  the  SJFHQ  and  other  US  Government  (USG)  organizations.51  A  stark  example  of 
agency  dichotomy  is  illustrated  between  the  Department  of  Justice,  Department  of  Homeland 
Security,  and  the  State  Department  concerning  a  new  fingerprint  database.  An  argument 
between  the  agencies  erupted  over  standardizing  the  number  of  fingers  per  hand  required  for 
the  prints.  Neither  agency  could  agree  on  the  number  to  be  used.  According  to  DOJ 
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Inspector  General  (IG)  Glenn  Fine:  “Agency  infighting... undermined  this  effort  [emphasis 
original].  The  agencies  ‘have  different  sets  of  mission  objectives,  and  each  one  has  been  a 

53 

forceful  advocate  for  its  respective  position.’” 

This  is  especially  true  outside  of  the  USG,  where  NGOs,  academia,  and  think  tanks 
are  involved;  the  COE  core  for  ONA  input.  “Each  is  not  a  totally  disinterested  party  and  so 
achieving  balance”  is  required.54  Political  biases  among  academics  may  arise,  as  well  as 
potential  pandering  to  ensure  DoD  funding  consideration.  NGOs  have  their  own  personal 
histories  and  their  “own  ‘beef  with  working  alongside  the  military.”55  Desire  for 
independence  and  non-alignment  may  prevent  certain  organizations  from  working  with  DoD 
altogether  or  cloud  the  information  provided.  Even  within  some  operational  missions,  NGOs 
may  limit  or  slant  data  to  preserve  their  neutrality.  Each  agency  or  organization  will  have  a 
specific  “solution  space”  they  can  provide  for  analysis;  whether  that  “space”  is  fully 
exhausted  will  affect  the  reliability  of  SoSA.56 

Value  weight  dissonance  among  different  SMEs  and  COEs  can  cause  problems  and 
requires  debate  among  the  ONA  analysts  and  the  collaborative  network  group.  Agency  or 
COE  contribution  can  affect  subsequent  course  of  action  (COA)  decisions  by  the 
Commander.57  Models  may  be  laden  with  information  “intentionally  misleading,  inaccurate, 
out-of-date,  and  incomplete.”  Faulty  assumptions  will  then  become  inherent  and  skew  any 
displayed  relationships  among  the  proposed  network  and  negatively  affect  results  that  will  be 
used  for  decision-making.  Information  used  for  SNA  modeling  must  be  critically  reviewed 
by  ONA  analysts.  The  goal  should  be  to  eschew  the  “quantity  is  quality”  factor;  the  number 
of  experts  consulted  does  not  a  fool-proof  simulation  make.59  Factor  weights  among  the 
Interagency  Community  and  COE  participants  may  differ  and  can  be  very  subjective.  Bias 
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and  analyst  perceptions  are  factors  that  cannot  be  adjusted  in  any  simulation  modeling 
process.  “You  can’t  just  wish  it  away  or  algorithm  it  outta  there.”60 

Social  Network  Analysis:  Limitations  and  Development 
Models  are  to  be  used,  not  believed .61 

—  H.  Theil 

In  order  to  conduct  analysis  to  determine  “what-if  ’  scenarios,  as  per  the  charge  for 
ONA,  we  must  look  to  social  and  business  decision  aids  as  examples.  "  System  simulations 

fx'X 

have  the  ability  to  test  various  policies  (actions  and  influences)  to  determine  effects. 
However,  the  analyst  and  the  warfighter  must  always  understand  the  simulation  is  nothing 
more  than  just  a  model.  It  is  not  ground  truth.  There  are  specific  limitations,  fully 
understood  by  programmers  and  researchers,  but  ignored  or  dismissed  as  assumptions  in 
ONA  doctrine. 

Current  systems  in  the  field  used  by  government  primarily  deal  with  traditional  social 
systems  comprised  of  small,  bounded  networks.64  There  are  problems  when  one  is  tasked  to 
run  analysis  upon  covert  networks  (such  as  a  terrorist  organization)  or  other  situations  of 
significant  missing  infonnation.  The  current  SNA  tools  do  not  scale  well  in  these  cases,  and 
grow  exponentially  flawed  due  to  error  with  increased  network  size.  There  is  no  “graceful 
degradation”  catch  within  the  algorithms.66  The  missing  data  can  be  somewhat  mitigated  by 
increasing  the  amount  of  true  knowledge  used;  however,  that  requires  many  specifics  (back 
to  the  “data  greedy”  concept)  and  can  be  extremely  difficult.66  Discriminating  data  in  order 
to  tailor  effects  and  results  within  an  AOR-wide  swath  at  the  Regional  Combatant 
Commander-level  can  be  increasingly  tricky. 
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Additionally,  the  current  DIME  construct  within  ONA  does  not  effectively  factor 
other  sources  of  US  national  power  that  can  affect  the  simulation  model,  such  as  SOF 

cn 

activities,  intelligence  collection,  humanitarian  assistance,  and  law  enforcement.  When 

viewing  through  the  lens  of  SASO  in  Iraq,  where  a  total  civil  culture  is  in  upheaval  and 

organizations  change  routinely,  simplified  simulation  runs  are  not  effective.68  The 

uniqueness  of  certain  crises  may  be  too  far  out  of  the  normal  “sim-box”  for  merely  allowing 

the  analytical  tool  to  run  calculations  and  offering  these  up  as  “the  answers.”  With  many 

variable  factors,  one  must  be  cautious  in  reliance  upon  the  network  as  a  capable  template  for 

analysis  under  every  situation.  Philip  Cerny  touches  on  this  premise  in  his  theory  of  a 

growing  “neomedievalism”  among  societies: 

As  in  the  Middle  Ages,  occupational  solidarity,  economic  class,  religious  or  ethnic 
group,  ideological  preference,  national  or  cosmopolitan  values,  loyalty  to  or  identity 
with  family,  local  area,  region,  etc.,  will  no  longer  be  so  easily  subsumed  in  holistic 
images  [my  emphasis]  or  collective  identities... National  identities  are  likely  to 
become  increasingly... divorced  from  real  legitimacy,  “system  affect,”  or  even 
instrumental  loyalty.69 

The  ability  to  maintain  accurate  computations  in  light  of  ever  greater  qualitative  change  is  the 
challenge  for  future  SNA  tools.  Assumptions,  inherent  within  current  modeling  tools,  must 
be  accepted  as  mere  possibilities,  not  pure  fact.  Simulations  “explore  realms  of  the  possible 
[my  emphasis]  rather  than  predicting  the  future  of  a  specific  event  or  action.”  Prediction  is 
difficult  and  can  be  dangerous  when  presenting  surmised  resultant  effects.  SNA  tools  cannot 
tell  us  what  Red  will  do,  but  they  can  tell  us  either  his  possible  reactions  or  those  effects 
negative  for  Blue.  Any  missing  data  or  uncertainty  will  degrade  the  prediction  as  those 
“holes”  are  extrapolated  throughout  the  model.  This  weakness  in  current  modeling 
techniques  is  understood  by  programmers.  As  an  after-action  discussion  following  war¬ 
gaming  with  JFCOM,  Dr.  Jim  Miller  of  the  Center  for  Adaptive  Strategies  noted  the 
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challenge  for  effects  detennination:  “Any  decision  support  system  that  is  reliant  on  such 
predictions  and  assessments  must  cope  with  rampant  uncertainty.”71 

Tools  are  needed  to  conduct  analysis  of  why  and  how  algorithms  compute  what  they 
do.  Given  the  previous  JFIC  example  of  20  man-hours  of  front-end  nodal  analysis 
crunching,  a  future  model  could  not  only  conduct  such  a  baseline  effort,  but  also  provide 
further  analysis  into  locating  higher  (or  lower)  confidences,  cueing  further  examination. 
Unfortunately,  such  tools  are  not  yet  fully  operational.  Work  in  this  endeavor  remains  within 
“pretty  new  and  untested  ground.”  Although  Joint  and  Fleet  advocates  for  ONA  and  SNA 
modeling  desire  a  snapshot  of  current  controlled  laboratory  systems  turned  into  a  tool  for  use 
now,  science  cannot  yet  support  this  level  of  capability. 

Therefore,  analysts  must  be  aware  of  the  dynamics  in  the  product  ultimately  packaged 
for  the  decision-maker.74  Analysts  and  planners  must  not  only  be  conscious  of  simulation 
and  data  limitations,  but  also  cognizant  of  the  level  of  error  or  model  adaptability  permitted. 
Because  ONA  cannot  only  be  focused  on  baseline,  long-term  data  analysis,  the  construct 
applied  must  be  scalable  to  support  the  commander’s  timeline  for  decision.  An  example  of 
such  a  scope  is  what  the  author  has  termed  the  “Butterfield  Scale,”  based  on  prior  study  of 
analysis  and  judgment  indicators  by  Alexander  Butterfield:75 


SITUATION 

TOLERANCE 

EXAMPLE 

Peacetime 

Low  tolerance  for  error 

Low  rate  of  change 

ONA  baseline  efforts 

Tensions 

Medium  tolerance  for  error 

Medium  rate  of  change 

Crisis  build-up 

Wartime 

Friction  accepted 

Metric  is  speed  of  assessments 

OIF  Phase  III 
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Understanding  the  cognitive  aspect  of  the  simulation  model  input  will  improve  the  capability 
to  discern  potential  “fault  lines”  within  the  results,  to  a  degree.  As  the  intensity  of  action 
increases,  simultaneously  with  the  desire  for  rapid  assessments,  scalability  must  come  into 
play  and  some  fidelity  tossed  over  the  side. 

The  “Butterfield  Scale”  allows  ONA  and  its  requisite  tools  to  remain  relevant  from 
major  combat  to  SASO.  To  make  the  process  relevant,  ONA  personnel  must  provide  “truth 
in  lending”  to  the  operators  and  the  commander.  Promising  peacetime  levels  of  granularity 
and  prediction  when  speed  of  dissemination  is  paramount  will  place  the  analyst  in  a  situation 
of  writing  checks  he  cannot  cash.  Many  PMESII  effects  require  a  significant  amount  of  time 
to  materialize  and  various  levels  of  ISR  support  are  necessary  to  coordinate  the  sensing  of 
those  effects.76  Along  the  scale,  one  must  determine  to  stick  with  a  proposed  model  and 
press  ahead  toward  production.  Once  that  decision  has  been  made,  the  analyst  is  obligated  to 
inform  the  planners  and  decision-makers  of  the  confidence  levels  for  the  analysis  in  a  final 
segment  of  ONA. 

The  Requirement  for  Critical  Debate 

This  type  of  knowledge  cannot  be  forcibly  produced  by  an  apparatus  of  scientific 
formulas  and  mechanics;  it  can  only  be  gained  through  a  talent  for  judgment,  and  by  the 
application  of  accurate  judgment  to  the  observation  of  man  and  matter.11 

—  Carl  von  Clausewitz 

In  review  of  business  management  approaches  to  decision-support  systems,  a  clear 
premise  is  founded  in  the  fact  that  “what-if  ’  gaming  cannot  replace  leadership  debate  over 

HO 

courses  of  action  or  intentions.  While  “[mjodcls  help  the  decision  makers. .  .understand  the 
key  mechanisms  of  an  existing  process”  by  revealing  a  specific  interpretation  of  relations,  the 
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presentation  to  decision-makers  must  encourage  debate  rather  than  provide  a  specific 
definitive  answer.  ONA  doctrine  does  call  for  a  review  of  effects  and  actions  for  each 
specific  DIME  source  of  national  power;  however,  self-analysis  is  limited  to  individual  teams 
focusing  only  on  their  particular  DIME  slice  of  the  overall  problem.  There  is  no  process  to 

“bring  it  all  home”  through  an  overall  analysis  to  compare  and  contrast  causal  results  among 
the  DIME  elements  against  each  other.  This  is  important  to  enhance  not  only  any  critical 
analysis  of  Blue  actions,  but  also  to  provide  various  possibilities  to  the  commander  on  Red 
intentions  and  response.  An  example  of  the  potential  for  differing  views  of  Blue  or  Red 
actions  lies  in  a  wargame  sponsored  by  JFCOM  in  2002.  Two  different  Red  teams  devised 
two  totally  separate  COAs  and  decision-processes  resulting  from  the  Blue  team’s  actions. 
After  review  by  CIA  and  DIA  representatives  present,  both  Red  COAs  were  deemed 
credible.  Scenarios  such  as  this  make  a  statement  for  ONA  teams  to  conduct  multiple 
reviews  of  action  proposals.  Further,  it  highlights  the  need  for  critical  debate  among  not  only 
ONA  and  EBO  planners,  but  the  commander  himself,  in  order  to  review  his  concerns  over 
various  possibilities. 

Analytical  output  of  ONA  and  EBO  cannot  be  presented  in  a  linear  fashion.  Products 
created  cannot  be  placed  before  the  commander  to  state  “ONA  says  X  will  result  in  Y.”  John 
Shanahan  presents  the  concept  of  “green,  yellow,  and  red  indicators”  for  prediction 
confidence.  This  compliments  the  “Butterfield  Scale,”  where  analysts  set  boundaries  for 
information  fidelity  based  on  operational  necessity.  Offering  a  range  of  possible  effects  or 
possible  influence  actions  along  with  confidence  values  allows  critical  debate,  not  just  a 
laundry  list  of  model  outputs.  To  simply  proffer  a  set  menu  would  be  akin  to  the  “worst  of 

83 

Vietnam-era  ‘systems  analysis.’” 
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Because  predictive  analysis  can  be  very  problematic,  if  not  outright  misleading,  one 
must  encourage  debate  among  planners  and  leadership  to  identify  acceptable  metrics  for 
decision,  whether  military,  social,  or  politically  motivated.  The  emphasis  must  be  exercising 
judgment  at  the  analyst,  planner,  and  leadership  levels.  “Most  important  decisions  inherent 
in  assessment,  planning,  and  operations  are  based  on  judgment  -  which  is  infonned  not  only 
by  infonnation,  but  also  by  discussion,  debate,  and  past  experience.”84  Debate  in  ONA 
literature  is  limited  to  EBO  planners  and  ONA  analysts.  There  must  a  broader  approach 
involving  command  leadership.  That  is  not  to  say  ONA  results  must  be  handled  in  an 
Industrial  Age  mentality.  The  SNA  tools  and  computational  systems  available  now  and  in 
development  can  be  significant  factors  in  an  intensive,  critical  analysis  of  EBO  options.  The 
need  for  scalable  analysis  demands  this  capability.  The  commander  directing  teams  to  argue 
with  the  computer,  based  on  their  individual  approaches,  to  discern  various  effects  and 
options  for  action  is  an  example  of  such  debate.  This  allows  both  the  commander  and  the 
analyst/planners  to  be  involved  in  the  process,  not  simply  forcing  the  commander  into  a 
“take-it-or-leave-it”  situation. 

Conclusion  and  Recommendations 

When  I  have  a  particular  case  in  hand,  I... love  to  dig  up  the  question  by  the  roots 

86 

and  hold  it  up  and  dry  it  before  the  fires  of  the  mind. 

—  Abraham  Lincoln 

Improving  ONA  requires  the  comprehension  of  shortfalls  existent  within  our  own 
assessments  and  predictions.  Bias,  error,  and  subjectivity  will  always  remain;  therefore, 
future  work  in  ONA  is  needed  to  readily  understand  thresholds  of  these  limitations,  provide 
degrees  of  confidence,  and  openly  engage  in  critical  debate  over  resultant  decisions.  Social 
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network  analysis  tools  used  in  SoSA  cannot  be  honestly  sold  as  the  sole  determinant  for 

on 

success.  Ideas,  systems,  and  metrics  are  moving  in  the  right  direction,  but  gaps  remain. 

ONA  theory  must  turn  away  from  the  “magic  bullet  syndrome”  and  instead  focus  on  limiting 
and  understanding  uncertainty.  While  analysts  cannot  fully  eliminate  preconceptions  and 
error,  they  can  leverage  effort  to  tamp  it  down.88  One  must  select  the  models  that  best  fit  and 
ignite  the  white  heat  of  analysis. 

JFCOM  must  engage  the  COE  for  SNA  research  in  concert  with  DoD  and  the 
Intelligence  Community  to  develop  more  adaptable  capabilities.  In  tandem  with  grappling 
the  mechanical,  ONA  teams  must  exercise  control  over  the  procedural;  viewing  not  as  a 
catch-all  menu,  but  rather  a  beginning  for  further  debate.  We  do  not  yet  have  reliable 
“devil’s  advocate”  analytical  systems,  and  work  is  needed  to  improve  SNA  analytical  tools 
for  military  decision-making  and  planning.89  A  realistic  ONA  process,  subsequent  to  a 
baseline  of  critical  self-analysis  and  validity  knowledge,  must  be  the  goal  for  future  research 
at  JFCOM  and  within  the  developing  SJFHQs. 
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